feat(node): probe foreign-chain RPC providers on startup by haiyuechen-nearone · Pull Request #3848 · near/mpc

haiyuechen-nearone · 2026-07-17T07:11:59Z

Part of #3764. Wires the identity-based check into the node. Merge the identity-probe stack (#3932 → #3962 → #3963 → #3964) first, then this trio bottom-up:

feat(node): probe foreign-chain RPC providers on startup #3848 (this PR) — probe every configured provider on startup + per-provider logs
feat(node): expose foreign-chain RPC provider health metric #3849 — per-chain configured / healthy Prometheus gauges
feat(node): surface healthy provider counts on /debug/node_config #3850 — healthy counts on /debug/node_config

Spawns a detached startup task that concurrently probes every configured foreign-chain RPC provider once — verifying its chain identity and inspecting a recent transaction, the same inspector and auth the real verification path uses — and logs a per-provider result plus an x/y providers healthy summary. The probe never blocks startup, and failure reasons aren't logged (they can carry secrets).

Expected identities come from foreign_chain_health_check.identities (no built-ins, so any network — including local — is checkable; a configured chain with no identity fails). ExpectedIdentities moves to node-config so the node and config-tester share one type.

claude · 2026-07-22T11:53:07Z

Pull request overview

Adds a startup health check that probes every configured foreign-chain RPC provider once against a golden reference transaction, then logs per-provider results and a x/y providers healthy summary. Runs in a detached task so it never blocks node startup. Mainnet/testnet always use a built-in golden set; local/sandbox chains only run the check when the operator supplies their own golden via a new foreign_chain_health_check_golden config field.

Changes:

New HealthCheckGoldenConfig (and BlockHashGolden/AptosGolden/SuiGolden) in mpc-node-config, plus a per-node foreign_chain_health_check_golden field.
Refactor golden.rs to represent the built-in mainnet/testnet sets as HealthCheckGoldenConfig values (unified type with the operator-supplied path).
New NetworkKind / HealthCheckRoute / SkipReason in network.rs, and resolve_network_from_config that trusts near_init.chain_id first and falls back to the contract-id suffix.
check_all_providers_with_golden runs the 10 per-chain probes concurrently via join_all, replacing the prior sequential mark_skipped / mark_not_configured mutation style.
New crates/node/src/foreign_chain_health.rs wires the route decision, logs results with secret-safe fields, and warns when a golden is set on mainnet/testnet (ignored).
New e2e test startup_foreign_chain_health.rs and mock-server plumbing (setup_evm_mock_with_block_hash, wait_for_log_substring) exercise the config-supplied golden path.
foreign-chain-config-tester now shares network_from_contract_id.

Reviewed changes

Per-file summary

File	Description
`crates/node/src/foreign_chain_health.rs`	New: startup entrypoint; picks a `HealthCheckRoute`, runs it, logs summary + per-provider status without leaking failure reasons.
`crates/node/src/run.rs`	Resolves network kind and spawns `run_startup_health_check` detached before indexer starts.
`crates/node/src/lib.rs` / `tests.rs` / `web.rs`	Module registration + test wiring for the new config field.
`crates/foreign-chain-health-check/src/lib.rs`	Adds `check_all_providers_with_golden`; converts per-chain runners to `Option<&Cfg>` + `Option<&Golden>` returning `Vec<ProviderResult>`; fans out via `join_all`.
`crates/foreign-chain-health-check/src/network.rs`	Adds `NetworkKind`, `HealthCheckRoute::decide`, `SkipReason`, `resolve_network_from_config`, `network_from_contract_id` and tests.
`crates/foreign-chain-health-check/src/golden.rs`	Reworks built-in reference tables to `HealthCheckGoldenConfig`.
`crates/foreign-chain-health-check/src/checks.rs`	Signature changes: `&str` params where owned refs used to be passed.
`crates/foreign-chain-config-tester/src/config.rs`	Shares `network_from_contract_id` with the node.
`crates/node-config/src/foreign_chains.rs` / `lib.rs`	New `HealthCheckGoldenConfig` + per-chain golden types; adds `foreign_chain_health_check_golden` to `ConfigFile`.
`crates/e2e-tests/src/cluster.rs` / `mpc_node.rs` / `foreign_chain_mock.rs` / `README.md`	Per-node golden plumbing, log-tail helper, EVM mock override for block hash.
`crates/e2e-tests/tests/startup_foreign_chain_health.rs`	New e2e verifying pass/fail per provider and skip-on-no-golden for a second node.
`docs/foreign-chain-transactions.md`	Documents the new startup probe behavior.

Findings

Non-blocking (nits, follow-ups, suggestions):

crates/node/src/run.rs:216 — The spawned health-check task's JoinHandle is dropped, so any panic inside a probe (e.g., a future bug in an inspector) silently disappears with no operator signal. Since this is diagnostic-only code the risk is small, but wrapping the future with FutureExt::catch_unwind (or spawning a small supervisor that logs on panic) would make failure modes debuggable. Consider aligning with the pattern used for _web_server_join_handle on line 210 if there's an established convention.
crates/node-config/src/foreign_chains.rs:63-85 — HealthCheckGoldenConfig field order (bitcoin, abstract, starknet, bnb, base, arbitrum, hyper_evm, polygon, aptos, sui) is neither alphabetical nor grouped by family. This will render in the same order in serialized TOML/JSON output and in the Debug impl. Alphabetizing would make config diffs easier to read and match the order in ForeignChainsConfig (if that one is alphabetized).
crates/foreign-chain-health-check/src/network.rs:90 — The mainnet contract-id fallback is a single hardcoded contract_id == \"v1.signer\" string. If a future mainnet deployment introduces another bare (non-.near) name, it will silently become Undetermined and skip the probe. Not a bug today, just a fragile piece worth a doc note or a small allowlist.
crates/foreign-chain-health-check/src/lib.rs:82-94 — Ethereum/Solana are hardcoded as "not yet supported by the node". When either becomes supported, the branch here needs to be dropped in lockstep with verify_foreign_tx/sign.rs; the existing referring comment mitigates but a TODO(#issue) linking the tracking issue (if one exists) would be safer against silent drift.
crates/e2e-tests/tests/startup_foreign_chain_health.rs:99 — The test asserts on the exact wording \"foreign-chain RPC provider health check complete: 1/2 providers healthy\", which pins the log string across two crates. Not wrong, but any future wording tweak in foreign_chain_health.rs will break this e2e; a contains on both \"providers healthy\" and \"1/2\" would decouple the check.
crates/node/src/foreign_chain_health.rs:83 — The TODO(#2350) comment is good; consider verifying that Define a systematic approach to secret redaction #2350 explicitly tracks "log health-check failure reasons after redaction lands" so the follow-up is not lost when Define a systematic approach to secret redaction #2350 gets closed on the general redaction work.

✅ Approved

haiyuechen-nearone · 2026-07-22T12:26:22Z

@claude

haiyuechen-nearone · 2026-07-22T12:42:59Z

+pub fn resolve_network_from_config(
+    near_init: Option<&NearInitConfig>,
+    contract_id: &str,
+) -> NetworkKind {
+    match near_init {
+        Some(near_init) => match near_init.chain_id {
+            ChainId::Mainnet => NetworkKind::Public(Network::Mainnet),
+            ChainId::Testnet => NetworkKind::Public(Network::Testnet),
+            ChainId::Localnet | ChainId::Sandbox | ChainId::Custom(_) => NetworkKind::Local,
+        },
+        None => match network_from_contract_id(contract_id) {
+            Some(network) => NetworkKind::Public(network),
+            None => NetworkKind::Undetermined,
+        },
+    }
+}
+
+/// Classify mainnet/testnet from a contract-id suffix.
+pub fn network_from_contract_id(contract_id: &str) -> Option<Network> {
+    if contract_id.ends_with(".testnet") {
+        Some(Network::Testnet)
+    } else if contract_id.ends_with(".near") || contract_id == "v1.signer" {
+        Some(Network::Mainnet)
+    } else {
+        None
+    }
+}


To reviewers: please double check that I have the correct understanding of our network and contract naming conventions

netrome

Started looking at this now, some initial thoughts:

We should not expect RPCs to retain individual TX information forever (blocking)
I'd prefer to minimize network-specific code paths. Ideally we should not have any network-specific code paths, but I can see the point of providing defaults for mainnet and testnet for convenience.

netrome · 2026-07-22T19:26:04Z

+    // A config-supplied golden is only meaningful on a local chain; on
+    // mainnet/testnet the built-in set is always used.


Why don't we provide golden configurations on mainnet/testnet? Hard-coding values feels like a time bomb, as many RPCs might not serve requests for old transactions.

My understanding was that we want to always check for the same transaction, which is baked in to the binary, maybe I misunderstood.

@anodar Judging from the foreign chain tester CLI code I assumed that we would like to continue using hard coded golden transactions for health check, maybe you had something else in mind?

netrome · 2026-07-22T19:29:39Z

+            "foreign_chain_health_check_golden is set in config but ignored on \
+             mainnet/testnet; the built-in golden set is always used — remove it \
+             from config.yaml"


Wouldn't it make sense to override the golden set if provided manually? The hard-coded values could exist as a convenience so we don't have to ask all operators to manually add these but retaining the ability to always override sounds like it would simplify our code. I'd prefer if we have as few network-specific code paths as possible. Ideally none.

netrome · 2026-07-22T19:31:53Z

+/// Hashes are hex, with or without a `0x` prefix.
+#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
+pub struct BlockHashGolden {
+    pub tx: String,
+    pub block_hash: String,
+}


I think this is a dangerous construction. Many RPCs won't serve transactions forever. It's better to ask for genesis hash or chain ID, like we do for Sui.

Spawn a detached, diagnostic-only startup task that concurrently probes every configured foreign-chain RPC provider against its expected chain identity and logs a per-provider result plus an `x/y providers healthy` summary — so config typos and un-enabled API keys surface at startup instead of on the first real verification request. The probe never blocks startup, and failure reasons are not logged (they can carry secrets). Expected identities are read from the node's typed config (`foreign_chain_health_check.identities`); `ExpectedIdentities` moves to node-config so the node and the config-tester share one type.

This was referenced Jul 17, 2026

feat(node): expose foreign-chain RPC provider health metric #3849

Draft

feat(node): surface healthy provider counts on /debug/node_config #3850

Draft

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 131a75f to e9e4b81 Compare July 17, 2026 08:03

haiyuechen-nearone force-pushed the 3764-extract-foreign-chain-health-check-crate branch from 05a55bc to ad30f88 Compare July 17, 2026 08:03

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 2 times, most recently from 4eeed22 to 8cdfbfa Compare July 17, 2026 08:35

haiyuechen-nearone force-pushed the 3764-extract-foreign-chain-health-check-crate branch from ad30f88 to a1145b2 Compare July 17, 2026 08:35

This was referenced Jul 17, 2026

Foreign chain RPC provider health checks: validate config without curated reference data #3764

Open

refactor: extract foreign-chain-health-check library from config-tester #3797

Merged

haiyuechen-nearone force-pushed the 3764-extract-foreign-chain-health-check-crate branch from a1145b2 to 9031612 Compare July 20, 2026 09:27

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 8cdfbfa to 36e439e Compare July 20, 2026 09:57

haiyuechen-nearone self-assigned this Jul 20, 2026

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 7 times, most recently from a2805eb to 3817516 Compare July 20, 2026 18:21

haiyuechen-nearone mentioned this pull request Jul 20, 2026

Run foreign-chain RPC health check on localnet (developer-supplied endpoints + golden) + seed provider whitelist #3860

Closed

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 30c77ea to 208c837 Compare July 21, 2026 09:24

Base automatically changed from 3764-extract-foreign-chain-health-check-crate to main July 21, 2026 09:52

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 208c837 to 7958f4b Compare July 21, 2026 11:30

haiyuechen-nearone marked this pull request as ready for review July 22, 2026 11:48

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 8e97b7e to 47f58d5 Compare July 22, 2026 12:32

haiyuechen-nearone commented Jul 22, 2026

View reviewed changes

haiyuechen-nearone requested review from anodar and netrome July 22, 2026 13:18

netrome requested changes Jul 22, 2026

View reviewed changes

haiyuechen-nearone marked this pull request as draft July 23, 2026 08:19

haiyuechen-nearone mentioned this pull request Jul 23, 2026

feat: probe Starknet by chain id and a recent transaction #3932

Open

netrome mentioned this pull request Jul 27, 2026

config tester: reuse the node's identity probe and drop golden transactions #3969

Open

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 485eb48 to 315514c Compare July 27, 2026 16:35

haiyuechen-nearone changed the base branch from main to 3764-foreign-chain-health-hybrid-aptos July 27, 2026 16:35

haiyuechen-nearone mentioned this pull request Jul 27, 2026

feat(health-check): probe Aptos by ledger chain id and a recent transaction #3964

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(node): probe foreign-chain RPC providers on startup - #3848

feat(node): probe foreign-chain RPC providers on startup#3848
haiyuechen-nearone wants to merge 1 commit into
3764-foreign-chain-health-hybrid-aptosfrom
3764-node-startup-health-check

haiyuechen-nearone commented Jul 17, 2026 •

edited

Loading

Uh oh!

claude Bot commented Jul 22, 2026

Uh oh!

haiyuechen-nearone commented Jul 22, 2026

Uh oh!

haiyuechen-nearone Jul 22, 2026

Uh oh!

netrome left a comment

Uh oh!

netrome Jul 22, 2026

Uh oh!

haiyuechen-nearone Jul 23, 2026

Uh oh!

netrome Jul 22, 2026

Uh oh!

netrome Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// A config-supplied golden is only meaningful on a local chain; on
		// mainnet/testnet the built-in set is always used.

Uh oh!

Conversation

haiyuechen-nearone commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Jul 22, 2026

Pull request overview

Reviewed changes

Findings

Uh oh!

haiyuechen-nearone commented Jul 22, 2026

Uh oh!

haiyuechen-nearone Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

netrome left a comment

Choose a reason for hiding this comment

Uh oh!

netrome Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

haiyuechen-nearone Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

netrome Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

netrome Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

haiyuechen-nearone commented Jul 17, 2026 •

edited

Loading